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Abstract. In this paper we establish a connection between the associated variety of a representation 
and the existence of certain degenerate Whittaker functionals, for both smooth and K-finite vectors, for 
all quasi-split real reductive groups, thereby generalizing results of Kostant, Matumoto and others. 



1. Introduction 

Let G be a real reductive group with Cartan involution and maximal compact subgroup K = G e . 
Let A4 = A4(G) be the category of smooth admissible Frechet G- representations of moderate growth, 
and let HC = HC (G) = HC(g,K) be the category of Harish- Chandra modules (finitely generated ad- 
missible (g, i4T)-modules) . Here g denotes the complexification of the Lie algebra go of G, and analogous 
notation will be applied without comment to Lie algebras of other groups below. We will denote a typical 
representation in A4(G) by (ir, W) (or tt or W) and a representation in HC (G) by (a, M) (or a or M). 
By [Wall921 Chapter 11] or )Cas89j or jBKj we have an equivalence of categories 

(tt, W) h-> (tt hc , W hc ) :M^HC 

where (tt hc , W HC ) denotes the Harish-Chandra module of if -finite vectors in (71", W) . 

We assume throughout this paper that G is quasisplit. We fix a Borel subgroup B with nilradical N 
and 0-stable maximally split Cartan subgroup H = TA, and we define 

(1) n' = [n, n] , = n/n', * = 0* C n*, * = {ip G * : i' (x) G iM. for x G n } . 

Thus ^ is the space of Lie algebra characters of n or cquivalently, via the exponential map, group 
characters of iV, while corresponds to unitary characters of N. We say that ip is non-degenerate if its 
i?c _ orbit is open in "J. We define ^ x to be the set of nondegenerate characters, and set = ty* fl \l/o- 
For ijj € tt G A4 and a G HC we define the corresponding Whittaker spaces as follows 

(2) Wh%(n) := Hon$(7r,V0,*(7r) := W & * = Wh%(*) £ 0}, 

(3) Wh' 4 , (a) := Hom n (<7,V),*(cr) := {tp G * : Wh^ (a) ^ 0}, 

where Hom^ (•) denotes the space of continuous iV-homomorphisms (functionals). We also define 
* x (tt) = *(tt) n * x , * (tt) = *(tt) n * , *o M = n *o 
* x (a) = #(<r) n * x , * (?r) = n * , * X O) = *(°") n *o 

If (tt, W) G M(G) then JT HC is dense in W and thus 

WTi^Tr) C W% (7r ffc ) and *(?r) C $>(tt HC ). 

We say that tt or a is generic if \f x (•) is not empty. By jCHMl Theorem 8.2] we have ty x (tt) = (tt). 
In fact using the same argument one can show \P (tt) = "Jo (tt) , but we will not use this result . 
Let Af C g* denote the nilpotent cone, and define 

He =Afnt ± ,AT =NT)g* . 

To a representation tt or a one can attach invariants such as the annihilator variety, associated variety 
and wavefront set (see ^2.21 below) 

AnV (•) C Af, AsV (•) C Af e , WF (•) c iAf 
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The dimension of these invariants determines the size (Gelfand-Kirillov dimension) of the representation. 
We say that tt or a is large if its annihilator variety is all of M . A key result of Kostant [Kos78j proves 
that a representation is large if and only if it is generic. More precisely for tt 6 M(G) one has 

AnV (tt) = AnV (tt hc ) = Af <S=> AsV (ir HC ) = Me ^==> WF (tt) = iAf Q 

* X (tt) ^ <^> * x (tt) + <=> * x (^ ffc ) + 

A number of papers (e.g. jGW80[ lMat87[ IMat90[ IMat88j ) provide certain generalizations of }Kos78] to 
non-generic representations; namely, they consider functionals equivariant with respect to non-degenerate 
characters of nilradicals of other parabolic subgroups, often referred to as generalized Whittaker function- 
als. In this paper we study a different type of analog: we consider functionals equivariant with respect 
to possibly degenerate characters of the nilradical of the standard Borcl subgroup. Following Zclcvinsky 
|Zel80[ §§8.3] we refer to these as degenerate Whittaker functionals. 

1.1. Main results. 

Theorem A. Let pr n * : g* — > n* denote the restriction to n, then for a € T-LC we have 

*(cr) =pr„«(AsV (cr)) n 

This is proved in section [3] below. We now describe the connection between ^o(tt), ^>o(tt hc ) and the 
wavefront set WF(7r). Let H = TA be the maximally split Cartan subgroup of G as above, and let 
Fq C Gc be the subgroup of order 2 elements in cxpiao- By |KR71j F G normalizes G and thus we get 
an action tt \— > TT a on representations of G, and we define 

i = 0{7r a :aeF G } 

We emphasize that the set-theoretic notation means that we only sum over distinct transforms ir a of tt. 
Now Fq normalizes go and no, hence it acts on \&o and we have 

* (7T a ) = a • *(tt), * (5f) = F G • *(tt) = (J {a • *(tt) : a € F G } 

Theorem B. Let tt £ M and write a = tt ; then we have 

(4) * (7t) C WF(tt) n* C * (tF) = *o(cr). 
Moreover if G = GL n (R) or i/ G is a complex group then tt = tt and we have 

(5) W (tt) = WF(tt) n * = *o(o") = AnV(cr) n # - 

If tt is generic, then Theorem IBI follows immediately from |Mat92[ Theorem A] and Theorem [AJ We 
prove the general result by reduction to the generic case using the Kostant-Sekiguchi correspondence and 
the coinvariants functor C u , where u is the nilradical of a suitable parabolic subalgebra. 

Theorem iBl implies AnV(er) f~l ^o 3 ^a(o~) though the reverse inclusion can fail, as shown in section 
14.31 for the group U (2, 2). We conjecture however that for all quasi-split groups one has the equality 

(6) *o(7r) = WF(tt) n 

although the proof probably requires additional arguments of an analytic nature. 

We prove a stronger result if G = GL n (R) or if G is a complex classical group, i.e. one of the groups 

(7) GL n (C) , SL n (C) , O n (C) , SO n (C) , Sp n (C) . 

Theorem C. Let tt € M (G) and suppose one of the following holds: 

(1) G = GL n (R) , GL n (C) or SL n (C); 

(2) G = O n (C) , SO n (C), or Sp n (C) and tt is irreducible; 
then ^o (tt) cmd WF (tt) determine each other uniquely. 

The first case of Theorem O follows easily from Theorem [Bj since for the groups in this case, every 
nilpotcnt orbit intersects V^o- This enables us to strengthen several results from |AGSj . We note that for 
unitarizable tt, a weaker version of this theorem follows from |GS12l Theorem A]. 
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For the groups in the second case of Theorem [C] not every nilpotent orbit intersects however if ir 
is irreducible then AnV (it) is the closure of a single nilpotent orbit (see |Jos85j 'l. and this allows us to 
deduce the second case from the following result that may be of independent interest. 

Theorem D. Every nilpotent orbit O for a complex classical group is uniquely determined by O H ^ . 

If 7r is not irreducible then WF(7r) might be the union of several orbit closures, and as shown in (1181) 
such a union is not determined by its intersection with ^q. We also note that Theorem [D] does not hold 
for exceptional Lie groups and we describe all the counterexamples in section f6. 31 

Remark. Over p-adic fields, the associated and annihilator varieties are not defined but the notion 
of wave front set still makes sense (see |HCh781 IHow741 IRod75| ). In [MW87] . the authors give a very 
general definition of degenerate Whittaker spaces and prove that the dimensions of "minimally degenerate " 
Whittaker spaces equal the multiplicities of corresponding coadjoint nilpotent orbits in the wave front set. 

1.2. The structure of our proof. In section [2] we give several necessary definitions and preliminary 
results on filtrations, associated/annihilator varieties, Whittaker functionals, and discuss a version of the 
Casselman-Jacquet functor. 

In section [3] we prove Theorem [21 Let (a,M) be a Harish-Chandra module for G, then every good g- 
filtration on M is good as an n-filtration. This implies that AsV n (M) = pr n * [AsV B (M)] where AsV n (M) 
denotes the associated variety of M as an n- module (by restriction). We next pass to the commutative 
Lie algebra t> = n/n' by considering the module of coinvariants C (M) = C n / (M) = M/n'M. Since t> is 
commutative, AsV D (CM) = AnV D (CM) and we denote both by V (CM). Then as shown in Lemma 
13.0. 11 V n (CM) = Supp (CM), which further coincides with <3>(M) by the Nakayama Lemma (see ffOj) . 

If M is any finitely generated n-module, V D (CM) C AsV n (M) n "P, and our task is to prove that 

V D (CM) D AsV n (M) n 

This is not true for a general n-module V, indeed CV could even vanish; for example, let G = GL(3,M) 
and consider the identification of n with the Heisenberg Lie algebra (x,-4-,l) acting on V — C[x\. 
However, if M is a Harish-Chandra module it was proven by Casselman that even M/nM is non-zero, 
indeed he proved that C\n l M = 0. This implies that M imbeds (densely) into its n-adic completion 
M := lim M/n'M. Following [ENV04] we let J (M) = M^ 11 " denote the submodulc of fj-finite vectors. 
The functor J can be applied to both M and CM and we prove that 

V B (CM) = V D (J (CM)) = V B (C (JM)) . 

The first equality follows from the fact that CM and J (CM) are both dense in (CM)^ and hence 
have the same annihilator, while the second follows from the isomorphism J (CM) ~ C (JM) proved in 
Lemma [3.0.21 Moreover J(M) is finitely generated over n and glued from lowest weight modules, and 
hence we get (by Lemma 13. 0.3[) 

V„(C(JM)) = V„(JM) n 
This reduces the problem to showing 

v d (jm) n * d AsV n (M) n 

which we prove in section f3. 31 using the main result of |ENV04j that describes J(M) deformation of 
M. The description is in the language of D-modulcs, using the Beilinson-Bernstein localization. While it 
is not true in general that the operation of taking associated variety commutes with limits, but this was 
proven to be true for holonomic D-modules with regular singularities in |Gin86j . This implies the above 
containment and finishes the proof of Theorem IA1 

In section @] we first prove Theorem [B] The special case of large representations follows from |Mat881 
IMat92j . To reduce to this case, note that any unitary character ijj of N defines a parabolic subgroup 
P = LU such that ip is trivial on NC\U and non-degenerate on NDL. Thus we consider the [/-coinvariants 
of 7r, and we need to know when this space is large as a representation of the Levi subgroup L. For that 
purpose we use Theorem [A] We also use [SV00] that shows that the wave-front set corresponds to the 
associated variety via the Kostant-Sekiguchi bijection. We next use Theorem [B] to reduce the proof of 
Theorem [Cl to Theorem ID] In subsection 14. 41 we deduce from Theorem [B] a formula for the wave front set 



4 



DMITRY GOUREVITCH AND SIDDHARTHA SAHI 



of JJ-coinvariants of 7r, where U is the nilradical of a standard parabolic subgroup P of G, whose Levi 
part is a product of general linear groups. 

In section [5] we give several consequences of Theorem [Cj including applications to the theory of deriva- 
tives. In section \6\ we prove Theorem iDl using basic results on nilpotent orbits from CoMG93l ICar85] . 

1.3. Acknowledgements. We are grateful to Avraham Aizcnbud, Dan Barbasch, Joseph Bernstein, 
Victor Ginzburg, Anthony Joseph, Maxim Leyenson, Kari Vilonen, and David Vogan for fruitful discus- 
sions. Part of the work on this paper was done during the program "Analysis on Lie Groups" at the 
Max Planck Institute for Mathematics (MPIM) in Bonn, and we thank the organizers of the program - 
Bernhard Kroctz, Eitan Sayag and Hcnrik Schlichtkrull, and the administration of the MPIM for their 
hospitality. 

2. Preliminaries 

2.1. The Nakayama lemma. The classical Nakayama lemma is a commutative analog of the problems 
considered in this paper, and in this section we explain this point of view. 

Let A be a commutative algebra, finitely generated over complex numbers. The characters of A are 
ring homomorphisms A — > C. Such a homomorphism is uniquely described by its kernel, which is a 
maximal ideal in A. Conversely, by Hilbert's Nullstellensatz, every maximal ideal m G Max^4 is the 
kernel of a (unique) ring homomorphism (j> m : A — > C. For an A- module M and m G Max A, we have 
Hom J 4(M, 4* m ) = M/mM. Thus, we can define 

#(M) := {m G Max A | M ^ mM}. 

On the other hand, the support of M is defined to be 

Supp(M) := Var (AnnM), 

where Var denotes the variety of zeroes, and Ann M = {a G A | aM = 0} is the annihilator ideal of M. 

Lemma 2.1.1 (Nakayama). If M is finitely generated over A then Supp(M) = \&(M). 

For completeness, we deduce this result from the version in |AM69j . 

Proof. We have to show that for any m G Max A we have mM = M AnnM ^ m. But AnnM ^ 
m 1 G m + Ann M -£=>• there exists an clement of m that acts on M as identity. The last statement is 
equivalent to mM = M by jAM691 Corollary 2.5]. □ 

2.2. Associated variety and annihilator variety. In this section we let q be an arbitrary finite 
dimensional complex Lie algebra, and let U = U (q) be its universal enveloping algebra with the usual 
increasing filtration U % ,i > 0. By the PBW theorem the associated graded algebra U is isomorphic to the 
symmetric algebra S (q). For a q-modulc V, wc define its annihilator and annihilator variety as follows 

Ann (V) = {u G U : Vx G V ux = 0} , AnV(V) := Var (Ann V) C q* 

Here AxmV C S (q) denotes the associated graded space of Ann V under the filtration inherited from U. 

For the rest of the section we assume that V is generated by a finite dimensional subspace V . 

In this case we get a filtration V' 1 = U l V°. The associated graded space V is then an S (q) module 
and we define the associated variety to be 

AsV(V) := Var(AnnF) = Supp (V) C q* 

It is standard that AsV(V) does not depend on the choice of the generating subspace. More generally 
a filtration on V is called a good filtration if the associated graded space is a finitely generated S (q)- 
module, and any two good filtrations lead to the same associated variety. For a submodule W C V, a 
good filtration on V induces good filtrations on W and on V/W by W l = Wn V 1 and {V/Wf = V i /W i . 

Lemma 2.2.1. |Ber72j AsV(V^) = Var (Ann F°) where AxmV a = {u G U : Vx G V° ux = 0} . 

Corollary 2.2.2. We have AsV(V) C AnV(V), and equality holds if q is commutative. 
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If there is possibility of confusion we will write AsV q (V) etc. to emphasise dependence on q. 
If ro is a subalgebra of q we define the coinvariant space to be the quotient 

C m (V) = V/xoV 

If ro is an ideal then C W V is a q-module and the action descends to the quotient Lie algebra r :=q/ro. 

Lemma 2.2.3. If V is a finitely generated q-module then AsV q (C m V) — AsV z (C m V) C AsV q (V) n r*, 
where r* C q* in the usual way. 

Proof. Let Y° denote the image of the generating space V° under the quotient map V —> C m (V). Then 
Y° generates G ro (V) and Ann q Y° D Ann q V°+tt). The result now follows from Lemma \2. 2.11 □ 

As was discussed in the introduction, the converse statement is not true in general. 

2.3. Nilpotent orbits and wavefront sets. In this section we assume that G is a real reductive group. 
Let A^C g* denote the null cone, with Ne = NfM 1 - and No — NC\Qq as before. The groups Gc, Kc an d 
G act with finitely many orbits on N, Ne and No respectively. We write O' < O if O' is contained in 
the closure O of O, and we refer to < as the closure order. 

Theorem 2.3.1. There is an order preserving bijection between G-orbits on No and K^-orbits onNg. 

The bijection, which we denote by O H KS(O) = KSq(0), is called the Kostant-Sckiguchi correspon- 
dence |Sek87j - the order property is proved in [BS98] (see also |Ohta81j ). The group Fq C Gc acts on 
both Ng and No, and we have 

Proposition 2.3.2. If O is a G-orbit on No and L C G is a 9-stable Levi subgroup, then we have 

F G -KS(O) =KS(F G ■C),KS L (On [*) C KS G (C). 

If (ir,W) G A4(G) then W HC is dense in W and we can choose a finite dimensional A"-invariant 
generating subspace of W HC . It follows that we have 

AnV(Tr) = AnV(n HC ) C N, AsV(tt hc ) C Ne 

and the two varieties are unions of Gc-orbits and Ac-orbits respectively. 

There is a further invariant of 7r called the wavefront set, which was defined in lHow81j in terms of the 
global character of tt. This is a G-invariant set 

WF (tt) C iN 

which, by |Ros95al IRos95bj , coincides with the asymptotic support of tt introduced in |BV78j . As con- 
jectured in |BV78] and proved in jSVOOj one also has 

Theorem 2.3.3. If (n, W) G M(G) then WF(tt) = i KS (AsV (ir HC )) . 

It follows that for all tt G A4(G) we have 

(8) AnV (tt) = G c • AsV (tt) = G c • WF (tt) 

Now suppose that G is a complex reductive group, regarded as a real group. Then the real Lie algebra 
0o is already a complex Lie algebra, and we have q = Qo x q , and go is diagonally embedded into g. The 
Lie algebra 6 is also isomorphic to Qo, an d is embedded into q by X i— > (X, 9 (X)). For a nilpotent orbit 
O C N(g) we have O = 0\ x 2 where O t C N(go)- However, if O intersects ig^ C g* or t 1 - C g* then 
0\ = Oi, and thus O is defined by a single nilpotent orbit in g . By |Vog91| Theorem 8.4], only orbits 
intersecting t 1 - can be open orbits in the annihilator variety of an admissible representation tt G A4(G), 
and thus we will be only interested in such orbits. 
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2.4. Restricted roots and parabolic subgroups. Recall that H = TA denotes our fixed ^-stable 
maximally split Cartan subgroup. Let S and So denote the root systems of f) in g and do in g respectively, 
and let g a C g and Qq C go denote the root spaces for a £ E and /3 <G So- For a £ E let a denote the 
restriction of a to do then either a = or else a £ T,q. Moreover for any (3 £ Eo we have 

dim R ffl^ = \{a £ E : 5 = /3}| 

Every a 6 E is real- valued on ao and imaginary- valued on to- The involution 9 acts naturally on E and 
if a' = —9a then we have 

a'\a = a| ao ,a'| io = - (a'\ io ) 

Lemma 2.4.1. Let G be a real reductive group then the following are equivalent: 

(1) G is quasi-split. 

(2) For all a £ E we have 5^0. 

(3) For all (3 £ E , we have dim R (g^\ < 2. 

Proof. Since G is quasi-split iff go has a Borcl subalgcbra, the lemma depends only on the Lie algebra 
0o- Moreover it suffices to prove the lemma for simple factors of go. The result is obvious for split and 
complex factors, and by |He08j the other possible simple quasi-split factors are of the form 



00 




SU;,/+i 




*6(2) 


Label 


AIII (r = 21 - 1) 


AIII (r = 21) 


DI(r = l + l) 


EII 



Now the lemma can be checked using Table VI of |He08j . where (2) means that there are no black dots 
in the Satake diagram, and (3) means that each of the multiplicities m\ and mi\ is at most 2. □ 

Since in this paper we suppose that G is quasi-split, we obtain 
Corollary 2.4.2. If a £ E satisfies dimu (jjg) = 2, then a\i ^ and g a n Qq = {0}. 

Proof. Suppose by way of contradiction that a\t = 0. Since dimR (qq) — 2 there is a root a\ ^ a such 
that a\ ao = ai|o - Since a\t a = 0, a\ must be nonzero on to, and thus a, a\ and = — 9ot\ are three 
distinct roots with the same restriction 5, contrary to assumption. Hence a\t =/= 0. 

Since a|t is imaginary valued we may choose X £ to such that a (X) = i. Now suppose v £ g a D Qq. 
Then we have [X, v] £ [to,0o] C go while on the other hand [X,v] = a(X)v = iv £ iga- Thus we get 
iv = 0, and since v was arbitrary we conclude that g a n 0q — {0} ■ ^ 

Our choice of B determines simple roots II C E and ITo C Eo, and the restriction a is simple if a is 
simple. Let 0, *o be as before and define Do = no/ [ty), n o] 80 that = (t>o) c and ^o = «0q, where 
denotes the space of M-linear fiunctionals on Co- The natural projection n — > restricts to isomorphisms 

© Cfl") « «, © (flS) « tio 

aet /3et„ 

We write 3 C I) for the center of g, and 3 C C f) for the stabilizer of ?/> £ ^. We recall that ^ is said to 
be nondegenerate if its TJc-orbit is open. 

Lemma 2.4.3. For ^6$ ifte following are equivalent 

(1) ip is non- degenerate. 

(2) H = 3 

(3) V>| s = 7^ for all a £ II. 

Proof. We note that dim (vp) = dim (fj/3) = |II| is the semisimple rank of G, while the dimension of the 
_ffc-° r bit of ip is dim (h/s^ ; thus (1) is equivalent to (2). Also we have X £ 3 iff a (X) = for all a £ II, 
while X £ s.0 iff a (X) = whenever ip\ g a ^ 0; thus (2) is equivalent to (3). □ 

We now prove an analogous characterization for tf> £ V^qj using the following elementary result. 
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Lemma 2.4.4. Let Wq be a two-dimensional real vector space with complexification W , and let ui be a 
<C-linear functional on W that is real valued on Wo- Let W\ C W be a two dimensional real subspace 
such that Wq D W\ = ; then we have 

w \w Q = uj = uj\n\ = 

Proof. The first equivalence holds since u> is C-linear. Also clearly us = =>■ <jj\w-i = 0. Conversely 
suppose u}\wi = 0. Since uj is real-valued on Wo and diniR Wo — 2 we have ker w n Wo ^ 0. Since 
Wq H Wi = 0, this forces ker ui 3 Wi- Since dime W = 2 we get cj = as desired. □ 

Proposition 2.4.5. for ^ € £/ie following are equivalent 

(1) tp is non- degenerate 

(2) ^| fl g + for all /3 € n 

TTie if orftii of tp is open in "Jo- 
(4) s V; n f) = I n go- 

Proof. The equivalence of (3) and (4) follows from a dimension argument similar to Lemma 12.4.31 It 
suffices to show that (4) is equivalent to (2) of Lemma 12.4.31 which is obvious, and that (2) is equivalent 
to (3) of Lemma \2. 4. 31 For the latter it is enough to show that if a € II and /3 = a then 

(9) ^ = <=► iij\ a , = 

Now © is obvious if dimflQ = 1 for then g a is the complexification of Qq. Otherwise by Corollary 
I2.4.2l we have Qq D @" = 0, and (j9j) follows from the previous lemma with Wo = Qq, Wi = Q a ,uj = itp. □ 

The standard parabolic subgroups of G are those that contain B, and these correspond bijectively to 
subsets of YLo- Indeed every P D B admits a Levi decomposition P = LU with 6>-stable Levi component 
L D H, the group B n L is a Borcl subgroup of L and the corresponding simple roots for Oo in (o give the 
desired subset of IIo. 

Lemma 2.4.6. For tp € Vl/o there exists a standard parabolic subgroup P = LU such that ip vanishes on 
Uo and restricts to a nondegenerate character of (o PI no . 

Proof. Let P correspond to the set |/3 e IIo : ip\ s !3 7^ oj; then the result follows from Proposition 12.4.51 

□ 

2.5. Nilpotent orbits for complex classical groups. If G = GL (d, K) or if G is a complex classical 
group as in ([7]), then the real nilpotent orbits of G are naturally indexed by partitions, as in |CoMC93] . 
A partition A of d of length I is a weakly decreasing integer sequence Ai > A2 > • ■ • > A/ > such that 
Xj = d. The Ai are called the parts of A, and the number of parts of size p is called the multiplicity 
m p (A) of p. We write V (d) for the set of all partitions of d and V\ (d) (resp. V-\ (d) ) for the subset 
such that m p (A) is even for all even (resp. odd) p. We set Xj = if j exceeds the length of A, and we 
define a partial order on partitions as follows: 

A < n iff Ai H + Afc < \x\ + ■ • ■ + fXk for all k. 

Theorem 2.5.1. There is an order-preserving bijection between nilpotent G-orbits and the set V (G) 
below: 



G 


GL (d, K) , GL (d, C) , SL (d, C) 


0(d,C) 


Sp (d, C) 


V{G) 


V(d) 


Vi (d) 


V-x (d) 



The case of SO (d,C) is slightly different. We say that A G V\ (d) is "very even" if A has only even 
parts. Note that each even part must occur with even multiplicity, forcing d to be a multiple of 4. 

Theorem 2.5.2. The nilpotent orbits of SO (d,C) are the same as 0(d,C) except that the very even 
orbits 0\ split into two orbits for SO (d,C), denoted C{ and O 1 ^ . 

For proofs we refer the reader to |CoMG93] . especially Chapters 5 and 6. 
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2.6. The Osborne lemma. Let S(gY and U(q) 1 denote the usual nitrations of the symmetric and 
enveloping algebras of g and let I (g) = S (g) and Z(g) = U (g) s denote the subrings of g-invariants. 

Lemma 2.6.1. [Wall881 §§3.7] There exist finite dimensional subspaces E C S(q), F C U(q) such that 

S(gY C S(ny i EI( 9 )S(t), U( 3 y C U{xCfFZ{Q)U{t) 

As before let Af C g* be the null cone and let Afg = Af n 6 . 

Corollary 2.6.2. JTie projection pr n * : Afe — > n* is a finite morphism. 

Proof. The maximal ideals S' >0 (t) C S(t) and / >0 (g) C /(g) vanish on on t and A/" respectively, hence 
both ideals vanish on A/g. By Proposition l2.6.1l C[Afl] is generated by E as a module over S(n) = C[n*]. □ 

Corollary 2.6.3. If Z is any irreducible component of Me then pr n * (Z) = n*. 

Proof. By Corollary 12.6.21 pr n . is a finite map and thus its image is a closed subset of n* of the same 
dimension as Z. By |Vog91[ 8.4] dimZ = 1/2 dim (Af) = dim(n*), thus pr n * (Z) has full dimension, so 
pr n . (Z)=n*. □ 

Corollary 2.6.4 (Casselman-Osborne-Gabber). If a € HC(G) then a is finitely generated as a U(n)- 
module. Moreover, any good ^-filtration on a is good as an n- filtration, and every good b-filtration on a 
is good as an n- filtration. In particular, AsV n (c) = pr n « (AsV(er)). 

For proof of the "moreover" part see |Jos811 §7.8.1] or [AG Si Appendix B]. 

2.7. The Casselman-Jacquet Functor. As before let b = f) + n be the Borel subalgebra of g. For a 
b-module V we define its n-adic completion and its Jacquet module as follows: 

^ ^ / \ ^-finite 

V = V n = lim V/xCV, J (V) = J b (V) = [V n J 

We note that J (V) is different from the Casselman-Jacquet module considered in [Wal]. However it is 
closely related to the geometric Jacquet functor considered in |ENV04| . (See Theorem 12.7.61 below.) 

Let Q (b) be the category of finitely generated b-modules for which every good b-filtration is also good 
as an n-filtration. Note that Q (b) is closed under subquotients. The following result is due to Gabber. 

Theorem 2.7.1 ( |Jos81) . §7). If V £ Q(b) then we have f] k>0 n k V = 0. Hence V embeds into V with 
dense image. 

By the Artin-Rees theorem for nilpotent Lie algebras ( |McC67l Theorem 4.2]) we deduce 

Corollary 2.7.2. V ^ V is an exact faithful functor from Q(b) to the category of b-modules. 

Lemma 2.7.3 ( Jos85 , §3.5). If V £ G{b) then there exists a finite dimensional b,-invariant subspace 
Soo c J (V), which maps onto V/nV. 

Since this result plays a key role in the subsequent discussion, we include a proof here. 

Proof. Let fl,- be the set of (generalized) weights of f) appearing in xi^V/x^ +1 V . Then there exists i such 
that Vlj n flo = for all j >i. Let us define 

S= 0(V/nHO„. 

Then any (generalized) () -eigenvector of S can be lifted by successive approximation to a (generalized) 
b, -eigenvector of the same weight in V . In this way we find an f)-invariant finite dimensional subspace 
.Soo C V that maps bijectively to S and thus onto V/nV. □ 

Lemma 2.7.4. If V £ Q(b) and W C J (V) is a dense b,-submodule ofV then W = J (V) . 
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Proof. Let J (V) = \ Vj be the generalized f)-eigenspace for some fixed weight fj,. Then for all suffi- 
ciently large i we have natural ^-isomorphisms 

(10) j{vf « (v/xtvY « (y/vfV)" « W 

The first isomorphism follows since J (V)' 1 H ti ! V = for all sufficiently large i, the last by the density 
of W. It now follows from formula (fT0|) that J (V - ) = W. □ 

Lemma 2.7.5. J (V) is dense in V for any V £ G(b). Moreover V i— > J (V) is an ea;ac£ faithful functor 
from Q(b) to Q{b). 

Proof. Let be as in Lemma T2.7.3I and let Voo C J (V) be the n-submodule generated by Soo, then it 
follows that Voo £ G(b). Also arguing by induction on i we deduce that Voo surjects onto each n l V/ri l+1 V 
and hence that V x is dense in V. By Lemma r2.7.4l J (V) = Vx , and thus J (V) is dense and belongs to 
G(b). 

Corollary 12.7.21 implies that J is left exact. For right exactness, we need to show that if 4> : V — > V is 
a surjection then so is J(f>; since the image of J(f> is dense in V, this follows from Lemma \2. 7.41 Now to 
prove faithfulness it suffices to show V ^ implies J (V) ^ 0, but this follows from Corollary 12.7.21 and 
the density of J (V) in V. □ 

If M £ HC (q,K) then M £ Q (b) by Corollary 12.6.41 so the above results apply to M, indeed in this 
case Corollary 12.7.21 is due to Casselman. However one can say more. Let B = 9 (B) be the opposite 
Borcl subgroup, and let C (g, b) be the category of finitely generated g-modules, which are b-finite. 

Theorem 2.7.6. If M £ HC (g, K) then M is a Q-module and we have 

/ \ n-finite 

(1) J (M) = (M n J 

(2) J(M)eC( fl ,b). 

Proof. Part fll]) follows from [ENV041 Proposition 2.4]. More precisely |ENV04j proves 

(, — ^ \ n- finite ✓ , — ^ \ t)-finitc 
M s J = (m«J 

and we get part ([T]) upon replacing n by n. Part © is [ENV04, Proposition 2.2]. □ 

The theorem implies that AnV(JAf) = AnV(M), and that AsV(JM) is a union of B-orbits in 
AnV(M) nb 1 . It is known that dimAnV(AZ) n b = 1/2 dim AnV(M), but unfortunately this vari- 
ety has many irreducible components. We note also that As V B (JM) = As V n (JM) and thus from now on 
we will write just As V(JM). 

2.8. Whittaker Functionals. We recall that a representation in A4 or HC is said to be large if its 
annihilator variety is the nilpotent cone Af(Q), and generic if it admits a Whittaker functional for some 
non-degenerate tp G 

Theorem 2.8.1. For tt £ M the following are equivalent: 

(11) tt is larger tt is generic^ tt hc is generic. 
Moreover if tt is large and ip £ ^ is non- degenerate, then 

(1) Wh'^ (tt hc ) ± 0. 

(2) If ip £ then there exists a £ Fq such that Wh* a ,^{-K) ^ 

(3) Iftp^^o then Wh^(n) = 0. 

For tt irreducible the implications in ([TTjl are in |Kos78l Theorems K and L]. Part JT]) follows from 
jMat881 Corollary 2.2.2]. Part © is }Kos78[ Theorem K] and Part © is in jCHMl Theorem 8.2]. The case 
of general tt follows from this by exactness of the functors Wh^ and Wh'^ proved in jCHMl Theorem 8.2] 
and |Kos78[ Theorem 4.3] respectively. Using the notion of wave-front set, Part ^ can be strengthened 
as follows. 
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Theorem 2.8.2 ( [Mat92j . Theorem A). For tt g M{G), we have 

WF(tt) n * X = *o (vr) 

3. Proof of Theorem [A] 

Let n' = [n,n] and = n/n' be as in ((IJ, and for an n-module V, we denote the 0-module of n'- 
coinvariants by 

C (V) = C n , (V) = V/n'V 
Since is commutative AnV {CV) = AsV v {CV) and we simply write V {CV). We note that 

VeG{b) => C(V)6 5(b). 
For the rest of this section let M g (G) denote a fixed Harish-Chandra module. 

Lemma 3.0.1. We have 

#(M) = Supp B (CM) = V D (CM) . 

Proof. The Lie algebra f) contains an element h that acts by 1 on t>, and by the degree on S{v). Since the 
ideal Anns( ){CM) is f)-invariant, it is homogeneous and consequently Supp D (CM) = V B (CM). Finally 
*(M) = Supp D (CM) by Nakayama's lemma (see § j___3 - □ 

For the proof of Theorem [A] we need three further results, which are stated below and proved in 
sections I3.1I3.2I3."31 

Lemma 3.0.2. We have a b-module isomorphism C (JM) r. J {CM) 

Lemma 3.0.3. V D (C(JM)) = AsV n (JM) n 

Lemma 3.0.4. AsV„(JM) D AsV„(M) n *. 

We now prove Theorem IA"1 

Proof of Theorem\^\ By Lemma 13.0.11 and Corollary |2.6.4l we have 

#(M) = V D (CM), AsV n (M) =pr„.(AsV fl (M)) 

By Lemma T2. 2.31 we have V D (CM) C AsV n (M) n \P, and it remains only to prove 

(12) V (CM) D AsV n (M) n * 

By Corollary [HU A/ g G{b) and hence C (M) E G{b) as well. By Lemma J (CM) is dense in 
CM, since CM is also dense in CM we get 

V D (CM) = AnV D (CM) = V B (J (CM)) 
Now by Lemmas 13701 El and [31141 we get 

V D (J(CM)) = V (C(JM)) = AsV n (J(M)) n * D AsV„(M) n * 
This proves (fT2l and finishes the proof of Theorem [XJ □ 

3.1. Proof of Lemma 13.0.21 For V g t/(b) we let 1/ denote its n-adic completion and let J{V) = 

^\ ()-finite _____ 

V) denote the associated Jacquct functor as before. In this section we prove Lemma [3.0.21 in a 



more general setting. Let c C n be any fj-invariant ideal, and define C C {V) =V/cV. 
Lemma 3.1.1. For V € Q{b) we have C C J{V) m J{C C V). 

Proof. By Lemma 12.7.51 J is exact hence it is enough to show that cJ{V) — J{cV) as submodules of 
J{V). Since V dense in V, cV is contained in the closure of cV in V, which by the Artin-Rees theorem 
( |McC67[ Theorem 4.2]) coincides with tV . Since cV contains cV we see that cV is dense cV, and since 

^ / \ f)-finite 

J{V) is dense in V it follows that cJ{V) is dense in cV. Evidently cJ{V) C IcV) = J{cV), hence 

cJ(V) = J(cV) bv Lemma, I27Q1 □ 
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3.2. Proof of Lemma 13.0.31 We prove Lemma T3. 0.31 for a more general class of modules. As before, let 

fa = f) + n, n' = [n, n] ,t> = n/n',* = 0*. 

Let J (fa) be the category of fa-modules with a finite dimensional f)-invariant generating subspacc. Ev- 
idently if M is a Harish-Chandra module, or even from category Q (fa), then J (M) G J (b). Therefore 
Lemma 13.0.31 follows from the next result. 

Lemma 3.2.1. IfV €j (fa) then we have V v (C n >V) = AsV n (V) n 

Proof. By Lemma l2"T2"lil we have V v {C n >V) = AsV n (C n >V) C AsV n (V) n and so it suffices to prove the 
reverse containment. Let £ be a finite dimensional t)-invariant generating subspace of V , and let F be 
its image in C n i (V). By Lemma \2. 2.11 we have 

AsV n (CVV0 = Var(7) , 

where J is the annihilator of F in U = U (n) and J C S (n) is its associated graded space under the usual 
filtration U l of U. Therefore it is enough to prove that J vanishes on AsV n (V) n i.e. that if u € J 
then u vanishes on AsV n (V) n To prove this we need some additional notation. 

We fix p v G f) satisfying a(p v ) = 1 for every simple root a, and for an b,-module X we consider 
generalized p v -weights, which we refer to simply as weights. We write for the /Lt-weight space for 
jjL G C, and if a; is a weight vector we write [x] for the real part of its weight; thus [x] = Re (/x) for x € X M . 
This notation will be applied to U, V and to the filtrands U l and V' 1 = U l E. We also fix a weight basis 
vi, . . . , i> m of E, ordered so that [m] > [vj] if i > j. 

Now let u S J be as before, then u£ J DU d for some d, and since J is od (fj)-stable we may assume 
u € C/ ; d for some integer J. If Z > d then it € n'C/ d_1 and u = on all of therefore we may assume that 
Z < d. For 1 < t < m let L* C V denote the submodule generated by v-y,...,Vt. Since V is glued from 
the subquotients L'/L*^ 1 we have 

AsVn^^ljAsVn^VL*- 1 ). 

t 

Thus it suffices to show that u vanishes on AsV n (L*/L t_1 ) for each t, i.e. that 

uv t g l' -1 + y d_1 . 

Now we may write ™ t = X)t=i ^ij°ij v ^j > where X^j G n' and bij G Z7 arc weight vectors satisfying 

[uv t ] = [X i:j ] + [bij] + [vi]. 
We have [Xij] > 2, [u] = I < cZ, and [«t] < [vi\ for i > t. Thus we get 

[hy] = [u] - [X i:j } + [v t ] - [vt] <d-2fon>t. 

It follows that for i > t we have bij G U d ~ 2 and Xijbij G f . Hence we get 

t-l m 

w * = Z] x 'J h 'J r ' x 'J h 'J r ' e Lt ^ + v^ 1 - 

i=l j i=t j 

□ 

3.3. Proof of Lemma 13.0.41 We will use Beilinson-Bernstein localization |BB81j . the paper |ENV04j 
that describes the Casselman-Jacquet functor in geometric terms, and the paper [Gin86| that describes 
the behavior of the singular support of -D-modules under the nearby cycle functor. Let us describe the 
setting in detail. 

Let M be an admissible (g, K) module with infinitesimal character xa ; with parameter A chosen to 
be dominant. Then M is a (U\, if)-module, where U\ is the quotient of U (g) by the two-sided ideal 
generated by z — xx (%)■ Let V\ denote the A- twisted sheaf of differential operators on the flag variety 
X, then U\ = T(X,T>x). By a (V\, K) -module we mean a coherent XVmodulc that is if-equivariant. 
Such a module is necessarily holonomic with regular singularities. By Beilinson-Bernstein f |BB81j ) the 
global sections functor 

T : {(V\,K) -modules} -> {(U x , K) -modules} 
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is exact and essentially surjective, a section of T is given by the localization functor T>\ <E>u x (')■ Moreover 
if A is regular then T is an equivalence of categories. Let X\, . . . , X n be the -R'-orbits on X, and let TJ.X 
denote the corresponding conormal bundles. If Ai is a (T>\, A)-module, then its characteristic cycle (see 
[Gin86j ) is of the form 

n 

SS(M)^^2m l T* Xz X. 

i=l 

for some nonncgative integers nii. The characteristic variety CV (Ai) is the union of T^ X for which 
rrii > 0. Let us describe the connection between the characteristic cycle of a T> \-rt\odv\e Ai and the 
associated cycle of the Harish-Chandra module M := T(A4). Any point x £ X defines a Borel subalgebra 
b x C g. The tangent space T X X can be identified with g/b x and the cotangent space with (g/b x )* = 
(bx) 1 - C g*. This gives a natural embedding of the cotangent bundle T*X into the trivial bundle X x g*. 
The composition of this map with the projection on the second coordinate is called the moment map, 
denoted by p. By a result of Borho and Brylinski (}BB85j) we have 

(13) p,(CV(M)) = AsV 3 (A/) 
By Corollary 12.6.41 we have 

(14) AsV„(M)=pr n .(AsV B (M)) 

The paper |ENV04] gives a precise geometric description of J (M), that we now recall briefly. Actually 
jENV04j deals with J fi (M), so the description below is a trivial modification of [ENV04 . Let H be the 
maximally split torus of G and let /? v : G TO — > H be the cocharacter such that a o dp v = —Idc for every 
simple root a. By composing p y with the action of G on X, we get an action map a : Q m x X — > X. 
Consider now the following diagram 

X <A- G m xX^-A x xX-(^-{0}xXwX 

For a (T)\,K) module Ai, let $ (Ai) be the 2?A-modulc obtained by applying the nearby cycles functor 
to j*a* (M) along {0} x X w X. 

Theorem 3.3.1. [ENV04] $ (M) is a (V x , N) -module and one has 

T(<S>(M)) = Jn(T (.M)). 

In view of this theorem $ (Ai) can be regarded as the geometric Casselman-Jacquet functor. 

The paper |Gin86| describes the behavior of the characteristic cycle under the nearby cycle functor in 
the following way. For an algebraic variety Z, and a regular function / : Z C let U := / _1 (C \ {0}). 
Suppose we have an algebraic family St of subvarieties of Z t := / _1 (t) parameterized by t g C\ {0}. Let 
S C U denote the total space of this family and let S denote the closure of S in Z. Denote by limt_j.o 5* 
the algebraic cycle corresponding to the scheme-theoretic intersection S D / _1 (0) (cf. |Gin86 ', 1.4]). 

Theorem 3.3.2 f |Gin86j . Theorem 5.5). Let M be a V x -module over Z, let <S>fAi denote the nearby 
cycle functor and let it denote the embedding of f it) into Z. Then 

SS($ f (M)) = KmSS((it)*(it)*M). 

Proof of Lemma \3.0.4\ From Theorem 13.3.21 we obtain 

SS($f(M)) = lim p y (t)SS(M) 
and passing to characteristic varieties we get 

(15) CV($ f (M)) = limp w (t)CV(M) 

Identify n* with the subspace of g* consisting of vectors having negative weights under the action of 
dp v (l), [n, n] with vectors having weights at least —1 and ^ with those having weight —1. Then by (fTB")) 
and p^]) . AsV n (A J f)n x I' is obtained by intersecting CV(A4) with the constant bundle Ax [n, n]^, projecting 
to the second coordinate and then further projecting to ^. Denote this operation on subvarieties of T*X 
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by py. Since the characteristic variety is a conical set (in cotangent directions), p*(p v (t)CV [M.)) does 
not depend on t. Since X is complete we get 

P9(Vtmp v (t)CV(M)) Dp^{CV{M)) 

Thus we get 

P»(CV($/(M))) Dp*(CV(M)) 
Lemma 13.0.41 follows now from Theorem 13.3.11 □ 

3.3.1. A counterexample to a stronger statement. One might ask whether inclusion holds without inter- 
section with 'J', i.e. AsVn(M) C AsV(J(M)). The answer is no, as shown by the following example. 

Let G = GP(3, R) and let g be its complexified Lie algebra. Let b be the Borel subalgebra of upper- 
triangular matrices, let n be its nilradical, and let s be the space of symmetric matrices. Using the 
trace form, we identify g with g* and ia with n*. Let M be a degenerate principal series representation 
corresponding to the (2, 1) parabolic. Then we have 

AnV B (M)=n, AsV B (M) =TZC\s 

where 1Z is the set of nilpotcnt matrices of rank < 1. 

For a lower triangular matrix let a, 6, c denote its entries as shown 

"0 " 

a 
b c 

Then we get 

AsV„(M) = prn (nns) = {a 2 b 2 + a 2 c 2 + b 2 c 2 = 0} 
AsV n ( JM) CKnn = {ac = 0} 

4. Proof of Theorems [Bj O 

4.1. The Jacquet restriction functor. As before let B be the fixed Borel subgroup of G. Let P D B 
be a standard parabolic subgroup, fix a Levi decomposition P = LU and let u be the complexified Lie 
algebra of U. For (it, W) € Ai(G) we have a natural representation of L on the space W/uW where uW 
denotes the closure of uW in W. This representation is usually denoted by rp (it) and referred to as the 
Jacquet restriction functor, for simplicity we will write rp (tt) = np. Its main properties are summarized 
below. 

Theorem 4.1.1. |Wall881 3.8.2 and 5.2.3] 

(1) np e M{L) 

(2) rp is left adjoint to the parabolic induction functor Ai{L) — > M{G). 

(3) (tt p ) hc = C u (tt hc ) := ir HC /uir HC . 

Remark 4.1.2. By an unpublished result of Casselman, uW is already closed in W , and hence W/uW = 
W/uW =: C U {W). 

Recalling the definition of $ = (u/ti') etc. from ([TJ) we write to denote its dependence on 
G. For each standard parabolic P = LU we can regard $j, as a subset of ^>g as follows: ^ l ~ 
{ip G ^/q : ip\ u = 0}. It follows immediately that for ir € A4(G) wc have 

* (7Tp) = * (7T) n *L 

Lemma 4.1.3. For 7r g A4(G) anrf P = LP D P we ftave 

AsV(7T P ) C AsV(tt) n I*,WF (tt p ) C WF (tt) n (* 

*o (tt) = (J * X (n P ) , * (7r ffc ) = (J * X (tt? c ) , * (5r) = (J (tF?) 
p p p 
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Proof. The result on As V follows from Lemma 12.2.31 and by |SV00] this implies the result on WF via the 
Kostant-Sekiguchi correspondence. The results on (n) an d ( nHC ) follow from the definition of Tip 
and Lemma r2.4.6l For the result on (tt) = UoeF G ^° ( 7r<J ) we reca U the definition of Fq C Gc, namely 

Fq = {a G cxpiao | a 2 = l} 
Since ao C [ we have a surjection a i— > ap : Fq — > Fl, such that (a ■ tt) p = ap ■ irp. Thus we get 
*o (5r) = |J * (a • tt) = |J * X ((a ■ tt) p ) = |J * X (a P • n P ) = (J* >< (^) 

a£F G a,P a,P P 

□ 

Lemma 4.1.4. If iX G * f»en A G pr n »(KS(G • A)). 

Proof. By Corollary 12 . 6.31 vr„ * projects any irreducible component of Afg onto n*. This implies the result 
if A is principal nilpotent. For general A, we choose a standard parabolic LU such that A vanishes on u 
and is principal nilpotent on [ n n. The result now follows from Theorem l2.3.1l and Proposition ^. 3. 21 □ 

4.2. Proofs of the theorems. We first prove Theorem 151 

Proof of Theorem\B By Theorems EUD and |£HH for a11 p we have 

* X (ttp) C WF (ttp) n % * x (tt^ c ) = * x (tFp) . 
Taking the union over all P D £? and using Lemma 14.1.31 we get 

(16) * (tt) C WF(tt) n *o(7r ffC ) = *o (tt) • 
By Lemma l4*X4l by |SV00] . and Theorem El we get 

WF (tt) n * C pr n . (KS(WFtt)) = pr n . (AsV(tt hc )) = *(tt hc ) 
Since WF (tt) C *o , it follows that 

(17) WF (tt) n * C * {k HC ) 
Combining (ITBl and (|T7)) we obtain (TJ|. 

Finally if G is a complex group or if G = GL (n, R) then each complex nilpotent orbit contains at most 
one real orbit. This has two consequences. First by (JSJ) it follows that 

WF (tt) n * = AnV (tt) n * 

Second, since the group Fq permutes the real forms of a complex nilpotent orbit, it acts trivially on 
orbits and we get = ^(tt). Thus © follows from (gj). □ 

We next prove Theorem [Cl using Theorem ITU 

Proof of Theorem^ In view of (JSJ it suffices to show that WF(7r) is determined by (n) = WF(7r)n* - 
For the first case, this is straightforward since every G-orbit in WF(-7r) intersects ^o- m the second 
case, since tt is irreducible then by |Jos85j there is a complex nilpotent orbit O such that AnV (tt) = O. 
Since G is itself a complex group, Oq = O n 0q 1S a single G-orbit and we have WF (tt) = iOq. Thus it 
suffices to show that Oq is determined by Co H^o, which follows from Theorem [Dl below. □ 

4.3. Some remarks on Theorem |Bj The action of Fq is not very significant in Theorem [B] For 
instance, let -0 € WF(7r) n a nd choose a parabolic subgroup P = LU such that ip is a principal 
nilpotent element in I*. Then we have shown that WF(7rp) contains some (real) principal nilpotent orbit. 
The action of Fl C Fq is used to permute the real principal nilpotent orbits, but if G is classical then 
there are only 2 such orbits (since L is then a product of a classical group with GL ni ). 

We next give an example to show that (tt) = ^o(tt HC ) can be a proper subset of AnV(7r) n ^o- Let 
P « GL (n, C) K Herm n be the Siegel-Shilov parabolic subgroup of U (n, n) where Herm n is the space of 
n x n Hcrmitian matrices. Let tt be the corresponding unitary degenerate principal series representation 
considered by Kashiwara and Vergne in |KV79ai IKV79bj . As shown in |KV79aj tt decomposes into n+l 
constituents ttq, . . . , 7r„, all of the same Gelfand-Kirillov dimension is denoted TT n -i t i in [KV79aj ) . On 
the other hand, the complex Richardson orbit Oc for P contains n+l real orbits Oq, . . ., O n as well, and 
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by a result of Barbasch (see |MT07j ) the associated wavefront cycle of 7r is ^ I OA , i.e. all multiplicities 
are 1. It follows that the wavefront cycle of each 7Tj is the closure of a single real orbit, which after 
relabeling we may assume to be Oi. 

For the group U (2, 2), 0c consists of matrices with rank 2 and square 0. It contains three real nilpotent 
orbits 00,01,02 whose representatives are the respective block matrices 



I 2 




diag(l,-l) 




-h 




The group Fq preserves 0i, and permutes Oq with 02. It is easy to see that 0\ intersects "I*, while Oq 
and 02 do not. Thus we get that 

*(7r,f c ) n * = M^o c ) = (0o"u 0;) n * 

is not equal to 0c H * = AnV(7r ) n * - Hence ^ (ttq C ) is a proper subset of AnV(7r ) n * - 

Theorem [Bl determines only $>o(ir HC ), not ^(tt hc ). From Lemma [3.0. ll wc see that $>(ir HC ) is Zariski 
closed, so one might ask whether ^(tt hc ) is the Zariski closure of ^o(^ HC )- Using the arguments of 
the above two subsections it can be easily proven for split groups. However, this statement does not 
generalize to all quasi-split groups. The representation 7Ti provides a counter-example for {7(2, 2), and 
a degenerate principle series representation (i.e. sections of a line bundle on the projective space CP ) 
provide a counter-example for 513(C). This can be shown using Theorem [Al 

4.4. Wave- front set of the Jacquet restriction. Theorem [Bl implies the following proposition. 

Proposition 4.4.1. Let ir G Ai{G) and suppose P = LU is a standard parabolic subgroup such that L 
is a product of several GL ni factors. Then 

WF(ttp) = WF(tt) n l* for alln eM (G) 

Proof. If L is a product of GL ni then Fl acts trivially on representations of L and hence $(irp) = \& (ttp). 
Now by Theorem [Bl we get 

wf(ttp) n * = * (7Tp c ') => *o(tt hc ) n r d wf(tt) ntnr 

Since WF(-7rp) and WF(7r) n [* are L-stable, and every nilpotent i-orbit intersects 'J, we conclude that 
WF(7rp) D WF(7r) n [*. The reverse containment follows from Lemma 14. 1.31 □ 

It is very interesting for us to know whether this holds without the assumption on L. Obviously, 
Proposition 14 . 4 . ll cannot have a p-adic analogue. 

5. Applications to GL (n) 

Let G n = GL(n,F) with F = K or C, and suppose tt € M(G n ). ,AGS] gives several definitions 
for the derivative of 7r, inspired by the p-adic notion defined in |BZ77j . Here we will use the following 
definition. Let P n C G n be the mirabolic subgroup, consisting of matrices with last row (0, . . . , 0, 1), then 
Pn ~ G n -i k V n where V n ~ F n ~ 1 C P n is imbedded as the last column. If (r, V) is a representation of 
p n and £ is a character of ra we can consider the coinvariants 

Q ( T ) = VVSpanjr (X) v - £(X)v : v g V, X e t)„}. 

Let ^0 be the trivial character of n and let (1 be the character given by 

£1(2:1, . . . ,x n -i) ■= \/^TRex n _i. 

The normalizers of £0 an d i n G n -i arc G„-i and P n _i respectively, and hence C7^ (r) and C^ t (r) are 
representations of g„_i and p„-i, respectively. We write $(t) = | det | -1 / 2 ® (r) and we define the 
fc-th derivative of r to be the following representation of Q n -k 

B k {T)=C^ k -\T). 

By [XGS1 Proposition 3.0.3] if a G HC{G n ) then B k {a) = B k (a\ p J G HC(G n -k), i.e. B k (a) is admissible. 

Theorem [Cl allows one to calculate the annihilator variety AnV (-B fc (<r)) in terms of AnV(er). For 
simplicity we consider the case of G n = 67L(n,M), since the case of GL(n,C) is very similar. Note that 
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AnV (a) is a union of complex nilpotcnt orbits 0\ C g* = jjl (n, C) , which are indexed by partitions A 
of n as in section 12.31 Also the nilradical n consists of upper triangular matrices and for each partition 
A we consider the character 

^W = V=T(E J ^^ + i) 

where S\ is the index set of partial sums {Afc + • • • + A; : 1 < k < 1} with / = length (A); then we have 
4>\ G O x . 

Lemma 5.0.2. Let /i be a partition of n — k and let /iU k be the partition of n obtained by inserting the 
part k in the appropriate place of ll. Then we have 

e * (B k a) i'^ k g * (a) 

Proof. Let a be the composition of n obtained by inserting the part k in the end of ll. It is a reordering 
of ll U k and thus ip a and ipfiuk belong to the same nilpotent orbit. The composition with the natural 
projection B k a -» a defines an isomorphism Wh'^ (B k a) = Wh'^ (a). Thus 

VV € (B k (j) -t=^ ip a €$ (a) <^> tl^ uk g * (cr) 

□ 

If A is a partition and k < Ai then there is a unique i such that Xi > k > A.; + i, and we define 
B k (X) := (Ai, . . . , Ai_i, A i+2l . . . , Aj) U (A,- + A l+1 - fc), B k (0\) = O b h {x) . 
We extend this definition to unions of orbits, setting B k ((D\) = if k > Ai. 

Lemma 5.0.3. Lei X be a partition of n and let fj, be a partition of n — k, then ll U k < X if and only if 
k < Ai and ll < B k (X). 

Proof. We use the notion of transposed partition (A*)i = max{j|Aj > i} and note that 

(1) transposition is order-reversing. 

(2) (n U fc)' is obtained from /Lt* by adding 1 to each of the first k parts. 

(3) (B k (X)Y is obtained from A 4 by subtracting 1 from each of the first k parts. 

The lemma follows. □ 

Theorem 5.0.4. If a g HC(G n ) then AnV(B k a) = B k (AnV(cr)). 

Proof. Since ip\ G 0\ , by Theorem [C] and Lemma 15.0.21 we have 

0„ C AnV(B k a) & ^ g * (S fc a) ^ ufe g * (cr) 0^ C AnV(a) 

Thus it suffices to show that for any O, O^ufc < O ^ < B k (0); this follows from Lemma |5"IQ1 □ 

In |AGS| the depth of a is defined to be the maximal rank of a matrix A g AnV(cr) C gl n (C), which 
is identified with ilfa£„ xn (C) via the trace form. Note that if AnV(er) = 0\, then depth(cr) = Ai. 

Corollary 5.0.5. If a £ WC(G n ) then B k (a) = if and only if k > depth(cr). 

By fJGSl Corollary 4.2.2], Theorem © implies 

Proposition 5.0.6. Let \i be characters of GL ni and n := n\ + ■ ■ ■+n^. Let ir = \i X • • • X Xk £ A4(G„) 
6e i/ie corresponding induced representation. Then 7r /ias a unique irreducible subquotient r wii/i 

AnV(r) = AnV(7r) = (ni ,..., nfc) . 

Moreover, r occurs in tt with multiplicity one. 

Proof. Without loss of generality, we can suppose n\ > • ■ ■ > rik and write A := (ni, . . . , nk)- Then it 
is known that AnV(7r) = 0\ and thus AnV(r) C 0\ for any subquotient r of it. Now by Theorem [Cl 
AnV(r) D Oa iff "0a € ^ (t) , and by |AGS1 Corollary 4.2.2], 7r has a unique such constituent. □ 
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6. The case of complex classical groups 
For convenience we fix an invariant form (x,y) on go and we identify iQq with go as follows: 

i>x (y) = i {x, y) . 

In this section we prove Theorem [D] 

Let H, B, IIo etc. be as before, then \I/o C go can be identified with the direct sum of negative simple 
roots spaces 



By Proposition 12.4.51 this identifies non-degenerate elements of ^ with the principal nilpotent elements 
in namely those for which each of the projections p@ : Qq^ is non-zero. Lemma 12.4.61 gives the 

following result. 

Lemma 6.0.1. J/e£$o then there is a standard Levi subalgebra Iq such that e is a principal nilpotent 
element in lo- 

We say that a nilpotent orbit O C go is a PL-orbit if O n lo is a principal nilpotent orbit in some Levi 
subalgebra Iq. Let PL (G) denote the set of PL-orbits, and for an arbitrary nilpotent orbit O we define 

PL (O) = {& < O | O' g PL (G)} . 

Lemma 6.0.2. For each nilpotent orbit O, the sets O n an d PL (C) determine each other uniquely. 

Proof. Let X denote the union of the orbits in PL (O) and let Y = O fl ^o- Then by the previous lemma 
we get X = G- Y and Y = X (1 * - □ 

Therefore Theorem [Dl reduces to the following statement. 
Theorem 6.0.3. For a complex classical group, every nilpotent orbit O is determined by PL(O). 
We will prove this in ^6.21 after describing the sets PL (G) for classical groups. 

6.1. Principal nilpotents in Levi subgroups. In this section we assume that G = GL (d, C) , O (d, C) 

or Sp(d,C), and write GL{d) etc. for simplicity. Nilpotent orbits for G are parametrized by partitions 
of d as in Theorem l2.5.11 we will regard PL (G) as a set of partitions and write PL (A) instead PL (0\) . 



be the partition corresponding to a principal nilpotent orbit; then 

Amax — 



(d - 1, 1, 0, . . .) ifG = (d) with d even 
(d, 0, 0, . . .) otherwise 

Proof. The principal nilpotent orbit is maximal with respect to the closure order. The result follows from 
Theorem 12.5.11 and the easy verification that A max is the maximal element in V (G). □ 

For a partition A write OM (A) = {p > 1 | m p (A) is odd} and define 

f V (G) = V (d) if G = GL (d) 

[ ' | {AeP (G) : \OM (A) | < 1} otherwise 

Proposition 6.1.2. If G = GL (d) ,0 (d) or Sp(d) then P L (G) = X (G) . 

Proof. For G = GL (d) the proposition asserts that every orbit is principal in some Levi subgroup, which 
follows from the Jordan canonical form. 

The Levi subgroups of O (d) and Sp (d) are given as follows: up to conjugacy there is one for each 
partition k with k% > • • • > K r such that d! = d — 2 (k\ + • • • + n r ) > 0. Explicitly 



L„ 



O(d') X GL{k x ) X ••• X GL(K r ) if G = O (d) 
Sp (d') x GL (m) x • • • x GL (n r ) if G = Sp (d) 
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X K = 



OM (A K ) 



The principal nilpotent orbit in L K can be determined by the previous lemma. In the partition A K 
for corresponding nilpotent orbit in Qq, each GL (/tj) factor contributes two parts of size K%. Thus up to 
decreasing reordering of the parts, we have 

(df — 1, 1, K\, Ki, . . . , K r , n r , 0, 0, ■ • • ) if G = O (d) with d even 
(d' , «i, Ki, . . . , K r , K r , 0, 0, ■ ■ • ) otherwise 

By definition, parts with even multiplicity do not contribute to OM (A) , thus 

OM((d'- 1,1)) if G = O (d) with d even 
OM ((d')) otherwise 

Moreover since the part 1 does not contribute to OM (A), we get \OM (A K )| < 1. Thus PL (G) C X (G). 

Conversely suppose A £ V (G) satisfies \OM (A)| < 1. Then A has 0, 1, or 2 parts with odd multiplicity, 
and in the last case the part 1 has odd multiplicity. Thus the last case can only occur if G ~ O (d) , and 
since there are exactly two odd parts with odd multiplicity, d must be even. It follows now that A is of 
the form X K for some k. Thus X (G) C PL (G). □ 

We prove Thcorcm l6.0.3l in the next subsection, using the following lemma. 

Lemma 6.1.3. For each A G V (G) and each k there is a partition fj, = /j, (A, k) € PL (A) such that 

Mi H h /ife = Ai H h A fc 

Proof. Let j be the largest index such that Xj = Afc. If (Ai, . . . , Xj) contains two or more parts p, q with 
odd multiplicity, then necessarily p 1 q have the same parity and so r = (p + q) /2 is an integer. If v is 
obtained from A by replacing a pair (p, q) by (r, r), then we have v < X and v\+ - ■ ■ + Vk = Ai + • • • + Afc. 
Iterating this we may assume that (Ai, . . . , Xj) contains at most one part with odd multiplicity. 

Now let fj, € V (G) be obtained from A by replacing the parts Aj+i, A^+2 • ■ • by a string of l's of length 
(A J+ i + Aj + 2 + • ■ ■ ) Then \OM (/i)| < 1 and hence [i satisfies the condition of the Lemma. □ 

6.2. Proof of Theorem 16.0.31 We now prove Theorem 1 6 . 0.31 for all classical groups. 



Proof of Theorem \6. 0.3\ First suppose G = GL (d) , SL (d) , O (d) or Sp (d) . We need to show that each 
A e V (G) is determined by the set PL (A). This is obvious for G = GL (d), SL (d) and therefore we may 
assume that G = O (d) or Sp (d). By definition of the partial order, for each k we have 

Hi H h Hk < Ai H h A fe for all [i e PL (A) . 

Moreover by Lemma 16.1.31 equality holds for some /i. Therefore for each k we can recover the sum 
Ai + • • • + Afc as the maximum of + • • • + fj,k for /i € PL (A), and hence we can determine A as well. 
Finally we consider G = SO (d). If O = 0\ where A is not very even, then O is a single O (d) orbit 



and so the result follows by the O (d) argument. If O = 0{ or O 1 ^ for some very even A, then O n "Jo is 



nonempty, thus O can be recovered from O (~1 \l/o m this case as well. 

The theorem does not extend to unions of orbits. 

Example 6.2.1. The table below lists examples of partition triples [A,/i,^] such that PL(X) 
PL(y). All orbits are special in the sense of Lusztig-Spaltenstein [CoMG93 s Section 6.3]. 



□ 



PL (fi) U 



(18) 



G 


A 




V 


0(11) 


(7,3,1) 


(5,5,1) 


(7,2,2) 


Sp (10) 


(6,4) 


(5,5) 


(6,2,2) 


0(8) 


(5,3) 


(4,4) 


(5,1,1,1) 



Remark 6.2.2. If G is a classical group, we can regard elements of go an d 0o as matrices. For a matrix 
X g 0\ its rank and order of nilpotence are given by n — length(X) and Ai respectively; we refer to these 
as the rank and depth ofO\. IfV is a union of orbits we define rank(V) and depth(V) by taking maxima, 
and the arguments above show that these are uniquely determined by V H \&o- 

For 7r <G M. (G) we define rank(n) = rank(WF n) and depth{ir) = depth(WFn). By |He08j rank{-n) 
coincides with the Howe rank ofn, and for GL(n), depth (j:) coincides with the notion of depth in section 
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It would be interesting to give a representation-theoretic characterization of depth for other classical 
groups. This remark shows that for all tt 6 M (G) the rank and the depth are determined by ^(tt). 

6.3. Exceptional groups. Theorem [Dl is false for every exceptional complex group and we now describe 
all counterexamples via the Bala-Carter classification |Car85[ §§13.4]. Let us say for simplicity that two 
nilpotent G-orbits O and O' are related if PL (O) = PL (£>')• 

Proposition 6.3.1. The following is a complete list of related orbits, with special orbits underlined. 



G 


Related orbits 


G 


Related orbits 


E 6 


Ee{di) and D§ 


G 2 


G2{a\) and A\ 


E 6 


D^iai) and A3 + A\ 


E 8 


E 8 (ai), E s (a 2 ) and E s (a 3 ) 


E 7 


Ej{a\) and E 7 (a2) 


E 8 


E 8 (a 4 ), E 8 (b4,) and E s (a 5 ) 


E 7 


£7(03) and Dq 


E s 


Ej(ai), E s (b 5 ) and E 7 (a 2 ) 


E 7 


Eq(oi) and £7(04) 


E 8 


Es(oq) and D 7 {a\) 


F A 


£4(01) and £4(02) 


E 8 


E^iai) and E 7 (ai) 


Fa 


F 4 (a 3 ) and G 3 (ai) 


E 8 


E 8 (a 7 ), E 7 (a 5 ), E 6 (a s ) + A x , D 6 (a 2 ) 



Proof. In Bala-Carter notation, the PL-orbits are labeled by the corresponding Levi subalgcbra [, while 
the other orbits have names of the form [(*). Thus for any orbit O we can easily compute PL (0) by 
looking at lower orbits whose Bala-Carter label has no parenthesis. With this in mind, the table above 
follows from the Bala-Carter classification tables |Car851 §§13.4] □ 
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